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Human Perceptual Model Applied to Rendering of Watermarked 

Signals 

Related Application Data 

This application claims the benefit of U.S. Provisional Patent Application Nos. 
5 60/480,993, filed June 23, 2003, and 60/463,175, filed April 15, 2003, which are hereby 
incorporated by reference. 

Technical Field 

The invention relates to digital watermarking of visual and audio data. 

Background and Summary 

10 Digital watermarking is a process for modifying physical or electronic media to 

embed a hidden machine-readable code into the media. The media may be modified such 
that the embedded code is imperceptible or nearly imperceptible to the user, yet may be 
detected through an automated detection process. Most commonly, digital watermarking 
is applied to media signals such as images, audio signals, and video signals. However, it 

15 may also be applied to other types of media objects, including documents (e.g., through 
line, word or character shifting), software, multi-dimensional graphics models, and 
surface textures of objects. 

Digital watermarking systems typically have two primary components: an 
encoder that embeds the watermark in a host media signal, and a decoder that detects and 

20 reads the embedded watermark from a signal suspected of containing a watermark (a 
suspect signal). The encoder embeds a watermark by subtly altering the host media 
signal. The reading component analyzes a suspect signal to detect whether a watermark 
is present. In applications where the watermark encodes information, the reader extracts 
this information from the detected watermark. 

25 Several particular watermarking techniques have been developed. The reader is 

presumed to be familiar with the literature in this field. Particular techniques for 
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embedding and detecting imperceptible watemiarks in media signals are detailed in the 
assignee's US Patents 6,614,914 and 6,122,403, which are hereby incorporated by 
reference. 

One challenge for watermarking visual media is maximizing the robustness, 
5 readability, and/or auxiliary information capacity of a digital watermark relative to a 

desired perceptual quality level. The digital watermark modifies the host image to embed 
auxiliary data, yet makes these changes using the assistance of human visual system 
modeling such that the changes are substantially imperceptible. While many have 
discussed perceptual image models that address this issue, several challenges remain, 
1 0 including effectively extending models beyond just luminance to address human 
perceptual models for colors, ^^and addressing the impact of printing to the perceptual 
model. 

This disclosure describes a method for digitally watermarking visual media taking 
into account a model of the rendering device, such as the printer or video display, to 

1 5 account for the effect of the rendering device on visibility of the digital watermark. In 
one implementation, a reference image and watermarked image are passed through a 
model of the rendering device and then to a perceptual quality metric to compute a 
visibility metric for the watermarked image relative to the reference. The digital 
watermarking process repeats this evaluation iteratively, while adjusting the embedding 

20 of the digital watermark to achieve a desired visual quality. 

In the case where the rendering device is a printer, the model of the output device 
takes into account such effects as the blurring effect, the tonal compression effect, and/or 
the dot gain effect of the printing process for a particular printer and substrate on which 
the image is printed. A similar approach can be applied to display devices, such as video 

25 monitors, video projectors, high definition video monitors. Plasma displays, etc. to take 
into account a model of the rendering device. As such, the method applies to still images 
and video content. 
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The technique can also be applied to watermarked audio, where the rendering 
device produces audio output and the system includes a model of the rendering device as 
well as an audio quality metric. 

In our implementation, we use the Watson metric as a visual quality metric, but 
5 other metrics may be used as well. 

Further features will become apparent with reference to the following detailed 
description and accompanying drawings. 

Brief Description of the Drawings 

Fig. 1 is a diagram of a perceptual model used in a watermark embedder to 
10 compensate for the effect of the rendering device on perceptibility of a digital watermark. 

Fig. 2 is a diagram of a visibility model used in a watermark embedder to 
compensate for the effect of a printer on the perceptibility of a digital watermark. 

Fig. 3 is a diagram of an iterative embedding process using a visibility model that 
compensates for rendering effects to adjust the embedding strength in iterations of the 
1 5 embedding process. 

Fig. 4 is a plot illustrating an example of spatial visibility distribution used to 
illustrate how a visibility map can be used to adjust the strength of a digital watermark in 
areas of an image. 

Fig. 5 is a plot illustrating measured print luminance vs. digital luminance for 
20 different rendering processes, namely printing on gloss paper vs. printing on newsprint. 

Detailed Description 

Introduction 

We have developed a system for embedding digital watermarks in media content 
25 while taking into account the impact of the rendering device on the perceptibility of the 
watermark in output produced by that device. Fig. 1 is a diagram of a perceptual model 
used in a watermark embedder to compensate for the effect of the rendering device on 
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perceptibility of a digital watermark. The perceptual model 18 receives reference and 
watermarked signals 10, 12. It then models the effect of the rendering device in a 
rendering model module 14. A perceptual metric module 16 then computes a 
perceptibility map 20 that is used to adjust the digital watermark selectively in areas of 
5 the watermarked signal that can tolerate more or less watermark signal strength for a 
desired amount of perceptibility. For example, the watermark signal strength can be 
maximized across the watermarked signal by varying it to maintain a desired 
perceptibility level as measured with the perceptibility metric. 

In the following description, we focus on visual content and rendering of such 
1 0 content through printers or displays. Similar approaches can be used for audio content 
using human auditory system models and models of the rendering device (e.g., D/A 
conversion and speakers) to compensate for the effect of the rendering device on the 
perceptibility of the watermark. 

Color Image Model for Rendering of Watermarked Visual Content 

15 Perceptual models have been applied to the watermarking of still images. Many 

of these methods use luminance based models that do not take into account the reduced 
sensitivity of the eye to high spatial frequencies in the blue-yellow and red-green 
directions. Most of the models also ignore the effect of the output process on image 
quality. For example, in offset printing the same digital image will appear very different 

20 if it is printed on newsprint with a 85 line screen ruling, compared to printing on high 
quality gloss paper with a 150 line screen ruling. Similarly images and video appear 
different if rendered on different display devices such as projectors, televisions, computer 
monitors, etc. with different display formats and parameters. 

To illustrate our method for offset printing, we describe a color image appearance 

25 model for Standard Web Offset Printing (SWOP) cyan, magenta, yellow, black (CMYK) 
data that is being watermarked. This model is used to calculate a grayscale visibility 
difference map. The visibility difference map is used to equalize the visibility of the 
wateraiark across the image, resulting in the maximum watermark signal at any visibility 
level. 
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This implementation of the color image appearance model is based on the Spatial 
CieLab and Watson metric, together with a model of the offset print process that is 
applied to the input images. Many factors affect the image quality of offset printed 
images. The major factors in offset printing that affect the watermark visibility are the 
5 paper stock and screen ruling. The model is used to automatically obtain the required 
visibility level for a watermark on characterized paper stocks and screen rulings. 

Fig. 2 is a diagram of a visibility model used in a watermark embedder to 
compensate for the effect of a printing process on the perceptibility of a digital 
watermark. The visibility model 28 takes as input reference and watermarked images 30 

10 and 32 and produces a visibiUty map used to adjust the watermarked image. The 

visibility model has three main components: color conversion module 34 (e.g.. Spatial 
CieLab conversion) for converting the input images to a format for visibility analysis, a 
rendering device model 36 for modeling the effect of the rendering device, and a 
visibility metric module 38 (e.g., Watson metric) for computing a visibiUty metric for 

1 5 areas within the input images. 

In one implementation, the reference and watermarked images 30, 32 are passed 
through a Spatial CieLab conversion going from CMYK to luminance. For more 
information on Spatial CieLab conversion, see Xuemei Zhang and Brian A. Wandell, 'A 
Spatial Extension of CieLab for Digital Color Image Reproduction', Proceedings of the 

20 SID Symposiums (1996). Altemative color conversion modules may be used. Also, if 
the visibility metric operates on color images, the input color images are converted to the 
color format used in the visibility metric module. For example, if the images are in 
CMYK format, they are converted to RGB format for a visibility metric module that 
operates on images in RGB format. 

25 In the case of offset printing, the luminance images are passed through an offset 

print model to correct for paper type and screen ruling, and the two images are then 
compared in the Watson metric to produce an output visibility display map as a grayscale 
image. For more information on the Watson metric, see U.S. Patent No. 5,629,780, 
which is hereby incorporated by reference. 
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The color image appearance model can be used iteratively within the watermark 
embedding software to obtain a marked image with the required level of visibility, on 
different types of print media from high quality offset on a heavy gloss paper stock to 
newsprint. The process of embedding a watermark in a CMYK image for print makes use 
5 of an original image along with a grayscale mask. Iterative embedding extends this by 
using the visibility map produced with the color image appearance model to create a new 
mask after each embedding iteration, as shown in Figure 3. 

Fig. 3 is a diagram of an iterative embedding process using a visibihty model that 
compensates for rendering effects to adjust the embedding strength in iterations of the 
10 embedding process. The embedder 54 receives as input an original image 50 and a mask 
52 used to control embedding strength in corresponding areas of the original image. It 
produces a watermarked image 56. As explained above, the visibility model 58 uses the 
original un-marked image 50 and the watermarked image 56 to produce a visibihty map 
60. 

15 In the iterative process, areas of the mask are adjusted up and down based on the 

visibility map and the previous mask to change the amount of watermark signal being 
embedded in different parts of the image as shown in block 62 of Fig. 3. The new mask 
64 is then fed back into the embedder 54 for the next embed. This continues until a 
visibility "goal" is reached, and the visibility map shows an even level of difference 

20 across the entire image. 

The embedding software automatically determines the maximum watermark 
signal level that can be inserted while achieving the required visibility on these different 
media. The signal level is also adjusted to compensate for the different types of image 
content, and the effect of watermark resolution. 

25 The model described above was used to calculate the visibility of a watermark 

added to an image. The visibility display map was used to iteratively adjust the 
watermark strength, until the required level of visibility was achieved across the whole 
image (see Figure 3). This allows the maximum watermark signal strength for a given 
level of visibihty to be inserted. If the visibility is not approximately constant across an 
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image, the watermark strength will be limited to the strength at which one area of the 
image first exceeds the required visibility level (see Figure 4, line 42). Less watermark 
signal is added in this case than if the visibility is approximately constant (see Figure 4, 
dotted line 44). 

5 The same model was used to calculate the visibility of a watermark (strength 1) 

added to an image which was then output to 100 pound gloss paper with a 150 line 
screen, compared to the same image output to newsprint with an 85 line screen. The 
visibility of the added watermark was predicted to be about 50% on the newsprint than on 
the 100 pound gloss paper. The paper stock has a large effect on the perceived image 
10 quality, due to the different tonal range and spatial resolution of the different paper 
stocks. The difference in tonal range for the different paper stock is shown in Figure 5. 

The appearance model was used to calculate the watermark signal level (strength 
2) required to obtain approximately the same visibility on newsprint. The difference 
between the strength 2 watermarked image and reference image on newsprint was 
15 subjectively about the same, as the difference between the strength 1 watermarked image 
and the reference image printed on 100 pound gloss paper. 

The color image appearance model we described has been used to obtain 
approximately equal watermark visibility across an image. The watermark signal strength 
has been corrected for images output on different types of paper stock, to obtain 
20 approximately the same visibility. 

Controlling the watermark signal strength with this model, allows a more 
consistent signal level to be added to the image which takes into account the following 
factors: 

a) image content 

25 b) signal loss due to the paper stock and screen ruling. 


The model we have described can be altered for other physical output processes 
such as screen display by replacing the 'Offset Print Model' component with a * Screen 
Display Model'. Further, an attention model could also be used to determine areas 
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within an image where a viewer's attention is most likely to be drawn. The results of the 
attention model can be used to generate a map indicating areas to adjust the watermark 
strength up or down to allow a stronger watermark signal to be placed in the areas where 
the viewer's attention is less likely drawn, such as backgroimd regions of an image. 

5 Concluding Remarks 

Having described and illustrated the principles of the technology with reference to 
specific implementations, it will be recognized that the technology can be implemented in 
many other, different, forms. To provide a comprehensive disclosure without unduly 
lengthening the specification, applicants incorporate by reference the patents and patent 

1 0 applications referenced above. 

The methods, processes, and systems described above may be implemented in 
hardware, software or a combination of hardware and software. For example, the 
auxiliary data encoding processes may be implemented in a programmable computer or a 
special purpose digital circuit. Similarly, auxiliary data decoding may be implemented in 

15 software, firmware, hardware, or combinations of software, firmware and hardware. The 
methods and processes described above may be implemented in programs executed fi-om 
a system's memory (a computer readable medium, such as an electronic, optical or 
magnetic storage device). 

The particular combinations of elements and features in the above-detailed 

20 embodiments are exemplary only; the interchanging and substitution of these teachings 
with other teachings in this and the incorporated-by-reference patents/applications are 
also contemplated. 


